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INTRODUCTION 



CONFERENCE PROCEEDINGS HAVE BEEN AN ANNUAL FEATURE 
SINCE 1958. PRIOR TO THAT DATE, AN ANNUAL REPORT 
WAS GIVEN ABOUT THE CONFERENCE AFTER ITS CONCLUSION. 

EACH PARTICIPANT THIS YEAR SUBMITTED HIS REMARKS ON 
SPECIALLY DESIGNED PAPER. EACH PAGE WAS PHOTOGRAPHED 
IN ORDER TO PREPARE A PLATE SO THAT THE PROCEEDINGS 
COULD BE PRINTED AS SUBMITTED BY THE AUTHORS, WITHOUT 
EDITING BY THE CTA STAFF. 

CTA ASSISTANT RESEARCH EXECUTIVE, DONALD P. GLASER, IS 
CONFERENCE PROGRAM MANAGER AND JOSEPHINE BROCK IS 
CONFERENCE PROGRAM SECRETARY. 
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Preambles What Is The Ent ity? 

The anatomy of education has acquired a new entity which is 
being variously referred to as a mutation, an affliction or a new 
metabolism. This new entity has been labeled "accountability. The 
cinating thing about labels is that they are widely _ ^ed by per sons having 
diverse understandings and attitudes toward the entity which the labe 
represents. It is much like the attempts of the three blind men to 
describe the nature of an elephant from their single and separate per- 
ceptions derived from feeling the trunk, the ear, and a hoof. 

Although some may scoff and remark, "It's just a word that 
is a fetish for the moment, and it will fade away," it can't be denied a 
newspapers, textbooks, entire issues of professional journals and in- 
numerable regional and national meetings have nominated this entity 
(accountability) for prime focus and attention. What indeed is this entity 
that has been attached to the anatomy of education? Is it a newly identi- 
fied but always existent gastro-intestinal distress? I s ^ a muta ion 
growth such as an extra eye, heart or gland? Is it a carbuncle, boil or 
rash? Or is it a new homeostatic process of the organism? The biopsy to 
be presented does not assure a discrete clinical diagnosis, but it is 
hoped that we may consider the etiology of current behavioral manifestations, 

Old Wine In New Bottles 

There is substantial evidence in the recorded history of 
previous ordered societies that the abstract idea of accountability was 
recognized as a concern to the members of the society and the basis or 
organization and action. Concern for the development of the young and the 
means by which such should be attained can be identified in the tnbal . 
societies, the Greek or Roman civilizations, the feudal societies or the 
Deluder-Satan laws of the New England colony of Massachusetts when the 
nation was first organizing a new society. Although education was 
assigned to and performed by different persons (such as by the adults in 
the family, a tutor and a student on a log, or a work group), the society 
had expectancies for how the members of that society should develop and 
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contribute. Moreover, individuals and groups were responsible for that 
development j and if a member of the society did not develop as expected, 
there were even such strict accountability consequences as banishment of 
the individuals responsible from the society. The Deluder-Satan law 
(an accountability pronouncement) was an interesting one, for it explicitly 
stated the rationale and predicted consequences if the younger generation 
did not acquire the skills and knowledges from appropriate reading. In 
essence, that colonial society was saying that education of the young was 
the imperative to prevent them all from falling into the tentacles of the 
devil. Parents, teachers and the town council were to be held accountable 
(and pay a fine) if the desired developments of the young were not achieved. 

If such historical data is accepted, then one must conclude 
accountability is an abstract concept that has been and is present in any 
ordered society. Society holds its members responsible and accountable for 
specified individual development. The major variations in this concept 
appear to be related to the question, "Who is responsible for what?" This 
is particularly crucial as one views the institution or system of education 
that has emerged in the last 200 years. The reorganization of institutional 
roles (and responsibilities) has resulted in the system of public education 
being ambiguously and variously perceived and held responsible for a vast 
array of the observed developments of the population. Society grimaces 
from the pain of such troublesome problems as defective communications, 
antagonistic social and cultural values, economic distress, changing 
vocational demands, and the disappearance of previous roles of individual 
participation in the society. 

There is no doubt that society is reacting to this dis- 
comfort by saying someone or something is responsible! A natural avenue 
to explore is the means by which the population has learned to be as they 
are. Ergo - the educational system could or should be responsible. More- 
over, education must be accountable for what it produces. 

So for the figure of speech, "old wine in new bottles," I 
would submit that societal concern for its members’ development is " o l d 
wine," while "who is responsible for what?" is a current "new bottle" 
demand that there be accountability assignments to the institutions or 
systems of the society. The "new bottle" seems to be the label of 
accountability for education. However, the prolific discourses on the 
label (accountability) suggest that this new bottle for old wine holds all 
sorts of elements, medicines and substances. 

What’s In This Vessel Labelled Accountability? 

It is apparent from the literature and ongoing discussions 
that various members of our society (and our educational systems) believe 
the bottle labelled "accountability" may contain any of the following: 

1. A strong drug that will cure the educational organism of an 
acute illness. At this time the exact characteristics of the 
illness are not defined. 
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2# A tapeworm which* if inserted into the organism* will be capable 
of growth in segmental units that will measure the health and 
productivity of the system# 

invisible gas that is odorous and unpleasant but will gradually 
disappear in the atmosphere# 

4# A potent new vitamin that will create a new metabolism of the 
organism and greatly increase its productivity# 

5# A dye that* when inserted intravenously* will identify the living 
and productive vs# dead and non-productive cells of the organism# 

Of course there are a host of other concepts defined or 
imagined that are important because the individuals holding these opinions 
are convinced that their definition is what accountability means or holds 
in store for them. The disparity of these expectancies for what will 
result from the phenomenon of accountability contributes to the ambiva- 
lence and confusion in emotional and cognitive reaction# 

The emergence of accountability has not been limited to the 
variety of expectations; it has also been associated with the development 
of new mechanisms or systems by which it is to be attained# The following 
ari illustrations of the procedures or systems that have been suggested 
es the *&&& by which accountability may be explicated and realized: 

l# P.P.B.S. (Planning, Programming, Budgeting System) - a management 
technique first employed by the national defense effort, which was 
designed to identify the relationships between the quantity and 
quality of products and the costs of alternative methods of 
production. In the national congress, it was referred to as the 
procedure to insure ”more bang for the buck#" All of us are 
aware of the substantial efforts in process in this and other 
states to design such a system for education. Indeed a guide and 
accounting manual are being compiled and. field-tested at this 
time. 

2. PERT (Program Evaluation Review Technique) - a management tool 
designed to assist school personnel to describe and monitor the 
operation of a program or system# 

3# Systems Analysis - a variety of operational research techniques 
including computer simulation which will define and analyze the 
components of a system and their interactions# 

4# Performance Contracting — A school system specifies desired out- 
comes with a desired pupil population and enters into a contract 
with an agency for the provision of educational experience s that 
will produce the desired outcomes in the pupil population# 
Guaranteed performance contracts specify that the contractor will 
not be paid if the specified outcomes are not achieved. 

5# Turn-keying - the process whereby a program established and 
operated effectively by a performance contract is adopted by a 
school system and operated by its personnel# 



6. Program Auditing - the independent examination of an educational 
program or performance contract to verify results, processes, 
personnel and the contents of the activities. 

7* Education Vouchers - suggests a means by which parents could buy 
in a "free market % f. the particular school program they selected by 
presenting a voucher for tuition issued by a school district or 
government agency. The implementation of such a plan implies the 
availability of financial and program audits, standards of 
educational quality and evaluative data to potential parent- 
purchasers. 

8. Incentive Pay - the rationale suggests that teachers will be paid 
on the basis of the performance or achievement of their pupils. 

The political arena has produced a wide variety of studies, 
commissions and task forces to accelerate the design and implementation of 
techniques and systems which will provide the information data-base for 
accountability. The commissions directed to work on cost-effectiveness 
models, goals and evaluation, and budgeting and accounting systems are 
illustrative of the apparent conviction that if a plan and procedure can 
be developed, a legislative mandate will insure designation of responsi- 
bility and indices for cost-effectiveness accounting for the educational 
program. The proliferation of these commissions and task forces has pro- 
duced a new quest for _?Who shall be the one to whom the educators are 
ultimately accountable?" And the divergent ramifications go on! 

Problems. Chronic Pains. Symtom Complaints Associated 
j With the New Entity 

i 

i The under-statement of the year might be the observation 

; that the total professional community related to education does not 

j enthusiastically embrace accountability (as divergently understood and 

| discussed) • 

» 

| Easy acceptance, ambivalence or negative reactions appear 

j to result from imprecise answers to the following questions: 

j 1. What are the schools to be accountable for? For all aspects of 

j affective and cognitive student accomplishment and development? 

j Such implies that there must be explicit and detailed statements 

of the anticipated outcomes in behavioral terms that are sus- 
ceptible to observation or measurement. Most educators would 
I . observe that detailed specifications of anticipated outcomes do 

| not exist, and even those that do exist have not been accepted by 

1 the various community and school-related populations as relevant, 

needed or appropriate. 

2. Who shall be accountable? Cold logic would suggest that each 

person whose task it is to influence learning - teacher, principal, 
curriculum coordinator, counselor or whoever - should be held 
, accountable for precisely that part of the educational outcome 

j which he can effect directly through his own efforts . Ambivalence 
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to such specification is bom from a reflection upon the com- 
plexity of human behavior as well as the infinite interrelations 
of environmental effects. 

3. How shall accountability be established? Obvious to all is the 
for a method of relating input factors to process and out- 
come in a manner that will permit appropriate attribution of the 
outcomes to the various input elements. Accountability requir 
a comprehensive information system for providing reliable in- 
formation on the input, content, process and products of the 
educational program. 

L. By whom shall accountability be determined? The insist ence_that^ 
self-evaluation is biased has promoted the notion that independent- 
external auditors or evaluators are desirable. On the other hand, 
program participants have recurrently voiced suspicion about the 
relevance of the process, measures, and data treatments that will 
be used as a means of judging the effectiveness of educational 
programs. As frequently observed in programs of evaluation, x,he 
person being evaluated is concerned with the understanding, 
knowledge and criteria which will be used by the auditor or 
evaluator. 

Although these four questions are ostensibly simple and 
straightforward, they are (from our experience in deriving explicit answers) 
complex and time-consuming if they are to have enduring acceptance and 
application. This seems to be capsulized by Richard E. Schutz m his 
article Development in Education (1970), when he called attention to t 
great distance between "construct" and " construction" in education. He 
observed that publicity, desire, political pressure and potential are not 
sufficient to convert constructs into construction. It seems that efforts 
to this point in time in implementing accountability through performance 
contracts, PPBS or criterion-referenced measures of behavioral objectives 
are still largely near the construct end of the construct-construction 
continuum. 

The incomplete and generally non-functional status of 
accountability systems seems to stem from naive assumptions of how ideas 
or constructs are transformed into realities. Many assume that rhetoric 
and intellectual exercises related to definitions, issues and implications 
are sufficient to result in delivery of operational systems. In contrast, 
the transition from theory to practice is more a process than a pointy To 
me, a salient question iss How do you get an idea such as accoimtability 
into the bloodstream, nervous system, actions and interactions which ar 
fundamental to the initiatives, responses and procedural patterns of a 
school system? 

Felix Lopez has made some useful observations related to 
why efforts to implement accountability have been unproductive. He lists 
the following pitfalls: 

1. Unrealistic management and legislative mandates in time and scope. 
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2, Imposition of accountability measures on unwilling and uncom- 
prehending staff. 

3- Misconception by management that accountability is an end rather 
than a means. 

4» Installation of system without build— up of background and under- 
standing of the accounting policies and procedures. (What infor- 
mation will be used for what purpose.) 

5. Expectation of great accomplishment and parsimonious answers with 
over-simplified procedures and minimal resources. 

6. Low or unknown reliability and validity of the measures to obtain 
data on which the accounting of accomplishment is to be judged. 

The foregoing pitfalls are suggested by Joseph Mazur to 
have existed because accountability was perceived as an externally 
initiated pressure, conceived and popularized by individuals who are safely 
protected from responsibility for day-to-day operation of the schools. 
Political types have threatened that "others shall be made accountable for 
their actions," and the external threats of accountability have been 
maximized. Such perception is in sharp contrast to the conviction that an 
accountability system is a means by which planning, documentation, evalua- 
tion and re-planning of the educational program may be facilitated. 

What About the Measures of Change? 

The new growth has produced anxiety if not distress among 
some of our most knowledgeable colleagues in the field of measurement. This 
has resulted from the simplistic suggestion that to observe students' 
growth during an instructional period, the accounting of that growth may be 
derived by subtracting the pre-test score from the post-test score. 

Further concern is directed toward using such computations with normative 
scores such as grade placements, for judging the change of behaviors of 
individual students during relatively short instructional periods such as 
three, five or seven months. In the haste to have hard data, such essential 
considerations as the standard error of the measure, chance scores below 
the floor of reliable measurement and other forms of error variance have 
been ignored. 

Program participants, in perceiving the necessity to show 
change on whatever is selected as the criterion measure of account ability, 
have unfortunately occasionally adopted the position that "We'll show em; 
we'll just teach the kids the contents and answers to the test items." 

Such a reaction reflects lack of understanding that any test of necessity 
only represents a sampling of a very large content domain} and by such 
exclusive emphasis upon the test, the comprehensive content of the 
curriculum is abandoned for the limited samples of the test items. What 
is perhaps most regrettable is that, while a fallacious momentary gain may 
be shown (over a six-month pre- and post-testing with teaching to the test 
items), the lack of substantial, usable and enduring achievement is 
reflected in the subsequent year of instruction. 
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In the accountability mode, the relevance of the content 
sampled by the measuring device to the content and objectives of the 
educational program is a common concern* While there is frequent in- 
sistence that nationally standardized measures should be used to observe 
change in student performance, there is nearly equal reaction that such 
tests may not accurately assess the particular contents and objectives of 
a particular school program* Criterion referenced measurement for the 
explicit objectives of a local program has been suggested as the solution 
to this problem* Unfortunately this is not an easy or quick solution. 

Roger Lennon has ably indicated that while there is much 
that can be said positively about such an approach it is not yet altogether 
clear how a series of criterion-referenced tests can be translated into 
units that will yield measures of gain or growth* Lennon also observes that 
criterion— referenced tests (of good qua li ty and scope) will be time-con- 
suming and costly to produce. Moreover, the methodology for the develop- 
ment of criterion-referenced measures is less well explicated than the 
development of norm-referenced tests. It seems clear that criterion- 
referenced measurement holds exciting potential as well as the need for. 
rigorous development in theory, methodology, instrumentation and analysis. 

Since accountability is dependent upon extensive, reliable 
and valid information, the concern for the manner in which. measurement is 
obtained is a critical professional problem* Time limitations of this 
brief biopsy prompt the suggestion that there must be increased study and 
understanding of the technical problems of measurement that have been dis- 
cussed by our professional colleagues such as Lord, McNemar, Stake, Wardrop, 
Lennon and Gronbach* 

Treatment and Nurture of Accountability 

It is not the intent of this analysis to produce a "great 
lament" or a new version of the "Perils of Pauline*" In contrast, it is 
my conviction that the idea of accountability as applied to the organization 
and conduct of an educational system is a venerable and potentially powerful 
approach to increasing the understanding, management and improvement of the 
educational program* This conviction directs us to look to the means of 
nurture that ‘may allow such a potential to be realized* 

Henry Dyer, who has devoted thoughtful attention to the 
analysis and conceptualization of the application of accountability in 
educational systems, has written extensively of the many facets and complex 
interrelations that must be recognized and planned for. Dyer suggests that 
the three interrelated aspects of accountability are concerned with legal, 
moral and informational considerations* 

The legal aspect of accountability has to do with the. means 
for ensuring that all the people in the educational system will act in 
accordance with the goals of the system. The focus of attention is upon 
effective action by all parties: voters, policy makers, administrators, 

teachers, specialists, parents and pupils. The legal aspect also refers to 
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the authority and contractual relationships by which the system operates. 

In the broad sense, all of the parties have a responsibility to some or 
several other parties. If the concept of legal accountability is to be 
a positive force in improving the quality of education, it is imperative 
that authority and contractual agreements are based upon sound educational 
goals and explicit agreements for the means of achieving the goals . 

The moral aspect of accountability has been defined by 
Dyer as ’’the ingrained habit of having care for others, particularly 
students, and being generally concerned for their growth and well being as 
persons.” While moral agreements cannot be legislated into belief and 
personal action, there is needed concern that contracts emphasizing methods 
and outcomes do not overlook or inadvertently destroy the moral account- 
ability of persons engaged in the educational process. 

The information aspect of an accountability program seems to 
be most fundamental to make the educational system functional. It is im- 
portant to emphasize that an educational system is a people system. It is 
a system made up of all kinds of people - policy makers, parents, teachers, 
administrators, pupils - all of them accountable for continally making 
decisions about how well the system is serving the pupils’ needs. As 
people are constantly making decisions, there is a need for everyone to 
have accurate and comprehensive information about the context, input, con- 
tent, process and outcome of the program. Indeed all of the several types 
of people related to the educational system must be held accountable for 
supplying, receiving and using information for decision making. Moreover, 
it is imperative that the exact nature of the information provided from 
parents, pupils, teachers or policy makers be available and understood by 
all if the multi-directional flow of interactions and decisions may con- 
tribute to an effective system. 

Dyer suggests that the three aspects of accountability are 
different ways of looking at the same unitary concept, and no one makes 
any educational sense unless the other two are also kept in mind. For 
example, everyone connected with an educational system has a legal and 
moral responsibility for providing necessary information to keep the system 
working effectively. At the same time, the information is necessary as a 
means of determining whether the people are fulfilling their legal and 
moral obligations to the goals of the system. Furthermore, there is ample 
evidence to suggest that legal directives are not likely to be accurately 
implemented and have much effect unless they are girded by moral commitment 
to the importance of meeting the educational needs of the children served 
by the system. 



The foregoing aspects of accountability which are operating 
in a people system must have clear definition and acceptance by all 
participants. If there are some participants who believe that by imposed 
authority or external mandate they are being held accountable for un- 
acceptable outcomes or pupil achievements that are beyond their influence - 
then the resultant "hang-up" by those concerned is the redirection of energy 
to oppose or neutralize the "external enemy" that is threatening the 

10 




15 



* 






internal family. This has been referred to as the "we and they syndrome." 
Typical comments of those perceiving such a condition would be: "It isn’t 

what I do; it’s the irrational demands or the problems ’they' create that 
cause all the difficulty." Of course the "they's" may be certain parti- 
cipants in the system, policy makers, or just an undefined population that 
inhibits the efforts of the "we" group. 

If accountability is to make a transition from concept to 
successful practice, it must be developed through the support of top leader- 
ship, but actively engaging all the related participants that include 
teachers, parents, policy makers, pupils, service personnel, etc. 

Joseph Mazur has suggested that, to initiate a system of accountability, it 
is necessary to create opportunities for the staff and other participants 
to be accountable. The first step in operationalizing accountability is to 
build readiness levels in each school by introducing, trying out and inter- 
nalizing the various elements of accountability. Such a thesis argues that 
a successful transition experience will be necessary to all personnel 
concerned with a school system so that accountability may produce better 
results than those obtained with present practices and attitudes. 

In addition to building acceptance of the construct and 
readiness, it will be necessary for each school system to find assistance 
in developing the capabilities for generating information for planning, 
development and evaluation. Among other things, this means that valid and 
reliable information must be obtained, summarized and communicated for the 
continuous process of decision making. The repeated reference to the 
essential need of comprehensive information or an educational accounting 
system also implies that the families or types of information must be coded 
or identified so that they may be related to the several aspects of the 
educational system. Even with automated data processing, files of data 
have commonly been accumulated that could not be interrelated to in- 
structional content, method, anticipated achievement, input characteristics 
or staff and facility conditions. This condition has formed a substantial 
obstacle to adequately describing the educational system and its functions 
in relation to observed developments of the student population. Recognition 
of this inadequacy has prompted Stuff lebeam to describe the C.I.P.P. model 
for evaluation, the conceptualization of C.E.I.S. (California Educational 
Information System), Dyer to create a Pupil-Change Model of a School as a 
Social System, etc. Figure 1 presents Dyer’s conception of the information 
concerning four groups of variables identified as input, educational process, 
surrounding conditions, and output. Input represents the entering 
characteristics of the pupil population with which the school program must 
cope. The pupil output is concerned with the attained accomplishment of the 
educational objectives. The surrounding conditions include all those 
factors in the school environment that may influence how the instructional 
program operates and pupils learn. Finally, the educational process 
variables include all the fac ili ties and activities in the school that are 
designed to accomplish the specified goals of the educational program. 

This model indicates the necessity for all variables to be 
measured and appropriately interrelated in various combinations to produce 
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interpretable indices by which the staff can observe how its own efforts 
are producing desired pupil change, after making due allowance for those 
variables over which it has no control. Such school effectiveness indices 
are conceptualized as being derived through a series of regression analyses. 



The Entity and the Enduring Purpose of Educ ation 

From this kaleidoscopic view of the entity labelled 
"accountability," we have viewed quickly the variety of attitudes and 
understandings that have emerged. Differential diagnosis of the existing 
condition or prognosis for the future health of the anatomy of education 
has not been attempted. 



The analysis of the symptoms, treatments and growths suggests 
that the construct of accountability holds high potential if it is viewed 
as an ongoing procedure that must be planned, initiated and implemented m 
education, which is a people system that should not be compared with an 
industrial operation dealing with inert materials. 



The statement by Dyer summarizes the foregoing conviction - 
"It must be constantly kept in mind that the educational process is not on 
all fours with a manufacturing or industrial process; it is a social 
process in which human beings are continually interacting with other human 
beings in ways that are imperfectly measureable and predictable. Education 
does not deal with inert raw materials but with living minds that are 
instinctively concerned, first with preserving their own integrity, and, 
second, with reaching a meaningful accommodation with the world around 
them. The output of the educational process is never a finished product 
whose characteristics can be rigorously specified in advance; it is an 
individual who is sufficiently aware of his own incompleteness to make him 
want to keep on growing and learning and trying to solve the riddle of his 
own existence in a world that neither he nor anyone else can fully under- 
stand or predict." 



I have enthusiastic commitment that we may make progress in 
furthering a central purpose of education, which is to increase our under- 
standing and rational means for decision-making and coping with our 
environment through effective and reliable information systems and 
accountability. At the same time, humility is suggested when we consider 
the dynamic and unique complexity of each individual in our society. As 
with an iceberg, there is much of human behavior and existence that is not 
externally visible. So as we progress to develop better systems for our 
educational systems, my hope is that we will also try and see the man or 
woman each individual is trying to be. 
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Dr. Gowan, when inviting me to participate in this 
conference, asked me to give "substantially the speech given at ETS" in 
Hollywood last March, "updated by the results you will have in hand in 
November." If more had been asked of me, I could not have complied. In 
addition to the fact that the evaluation was not completed until eight 
or ten days after the deadline for submitting this paper had past — not 
a very acco un table way for our evaluation team to perform -- enough crises 
have occured in Big "D" that accountability and performance contracting 

have not been given priority status of late. 

In fact the biggest D in Dallas lately is desegregation. 

On August .8, 1971 the Federal District Court gave its initial order to 
Dallas to implement a prescribed desegregation plan. Two amendments, two 
stays, and three clarifications later, we are still doing very little in 
Dallas but implementing orders, stays, and clarifications. We opened 
school two weeks late, and since that time, all available manpower has been 
riding buses, chasing down rumors, and answering telephones constantly and 
endlessly. We reassigned 7,000 teachers and twelve thousand students 
during a five day period. We have for several weeks now been working 
literally twelve to fourteen hours a day. 

We thought that the drive for accountability in Dallas 
brought about frustrations, but it was nothing to compare with the drive 
for desegregation. But perhaps our drive for desegregation will ultimately 
accomplish more for the disadvantaged student than all the compensatory 
programs we have attempted, including performance contracting, combined. 

For five years — ever since the passage of the Elementary 
and Secondary Education Act gave us the extra financial boost we needed to 
develop compensatory programs -- we had been trying all the old tricks 
and most of the new to improve achievement among those children whom we 
call the "culturally disadvantaged." Along with other school districts 
all over the country, we bought shiny new hardware and clever new software; 
invested in workshops and seminars for our teachers; sent our kids to 
concerts and museums and factories and even — courtesy of Braniff Airlines- 
up over the city in planes. In sum, we waved the banner of innovation as 
energetically as anyone. 

Naturally, even though we got a considerable boost from 
Title I and other forms of federal aid, our costs went up. They tripled 
in the last 10 years — mainly because of new construction, salary 
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increases, and improvements such as air-conditioning; but partially because 
we asked the citizens of Dallas to stretch federal dollars with their own. 
And when we totalled the results of this financial exertion on the part 
of the taxpayers, and of the spiritual exertion on the part of our teachers, 
we found we didn't have much to be proud of. 

Our target had been those schools in which students were 
averaging only a half-year's achievement gain for every full scholastic 
year. By the time we finished, we had not managed to improve on this sad 
record; in fact, some of our Title I schools were worse off in 1970 than 
they had been in 1965. 

Any sane school administrator is reluctant to hang out his 
dirty linen for public viewing. I cannot suppress a certain sense of 
embarrassment even now, as I speak. All that gives me courage to do so 
is the knowledge that virtually every other large city school system in the 
country has had the same experience as Dallas. Five years and five billion 
dollars after Title I was passed, we still have not learned how to break 
the cycle of under-achievement that sees children from poor homes do poorly 
in school; find poor jobs or none; marry — and then send their own poor 
children to school. 

But though this failure remains constant, some things have 
changed in education -- notably the public attitude toward those who run 
it. Ten years ago, we educators confidently asserted that we knew how to 
cure educational illness. All we needed was enough money to lower pupil- 
teacher ratios, put a library in every school, an overhead projector in 
every classroom, and so on and so forth. Our prescriptions for educational 
excellence were based on traditional notions that went unchallenged because 
a stingy public had never allowed us to try them. 

During the 1960's, we got a chance to try them; not as much 
of a chance as we would have liked, perhaps — too many school systems 
spread Title I funds around so thinly that the extra money could not have 
any impact. Nevertheless, we were given a reasonable chance — and the 
results did not justify the investment. And today, it is clear, the 
public does not believe it is getting its money's worth from public education 

Thus there is a public frustration as well as a professional 
frustration behind the drive for accountability. Public school systems 
have developed extremely precise methods of accounting; most of them can 
tell you to the penny how much they spent for teachers' salaries, textbooks, 
red mid blue litmus paper, and the wax on the gymnasium floor. 

But they cannot tell you what this investment produced . Our 
focus in educational accounting has been on input , not output . Professor 
Dwight Allen of the University of Massachusetts has quite properly 
criticized the accounting methods of school systems as being irrelevant 
for purposes of devising new educational strategy. Per-pupil expenditures 
do not really tell us \riiat it costs to educate a student; all they tell us 
is Tdiat it costs to keep a student seated for a year. 

A much more relevant measure. Dr. Allen argues, would be a 
"learning-unit" cost -- the total sum, including teacher's salary, portion 
of total building expense, cost of textbooks and other learning materials 
required to move a student from one skill- level in reading, writing, or 
math to the next highest level. These costs, moreover, would vary from one 
school to another; they would be higher in a school with a majority of 



children from low-income, black or Spanish-speaking families than they 
would be in a school with a majority of white children from upper-income 

homes. ,, , 

Developing such a new accounting system would enable edu- 
cators tp show the public how much learning was produced by a certain 
amount of investment. It would, moreover, enable educators to shift 
resources back and forth within a budget — testing, for example, the 
value of teacher-aides in one classroom against the value of educational 
technology in another and of programmed texts in a third. In each case, 
input would be related to output — and educators who prescribed various 
teaching strategies would be held accountable for the results they produced. 

Accountability is, in essence, a statement of policy. It 
states that educators will accept responsibility for their performance 
or lack of it. It implies that there is a contract between school 
personnel and the public, and that that contract involves more than showing 
up for work on time. It accepts the fact that culturally different 
backgrounds make the task of educating more difficult, but it asserts that, 
as professionals, educators can overcome — or will learn to overcome 
cultural difference. 

Now statements of policy are fine things, if for no other 
reason than that they look nice framed on a wall. But if a statement of 
policy is to be a genuine program rather than just a fashionable enthusiasm, 
it must be translated into a strategy --a set of practical steps for turn- 
ing an idea into a reality . 

Performance contracting is one such technique. It is not 
the only one. Voucher plans are another -- and so, indeed, is any systematic 
effort to relate educational effort to student achievement. 

Our sense of frustration In Dallas led us to try performance 
contracting. Our interest in it led us to two distinct programs — one 
financed by the Office of Economic Opportunity, and the other by Title I. 

I wish today to describe the Title I program because we controlled it from 
the start: chose the student population, outlined the performance criteria, 

wrote the request- for-proposals, defined the conditions under which any 
successful contractor would have to work, and negotiated the final contracts. 

The entire process has been carefully monitored by the mos t 
precise scientific methods. According to our most recent figures, for 
example, every administrator involved has lost an average of 13.1 pounds, 
given up 46.3 percent of his weekends, and antagonized 75 percent of his 
wives to the point where 100 percent of them threatened to go home to 
mother an average of 3.4 times. Nevertheless, innovation marches on in 
Dallas • 

First, a note on the OEO program. It involved about 600 
students in grades one through three and seven through nine in two schools; 
these students are matched with another 600 in a control group. The 
subject areas are reading and mathematics, both of which were subcontracted 
by OEO to Quality Education Development, Inc., of Washington, D.C. 

Contracts for two service components, audit and management support — I’ll 
explain these terms a little later on — were awarded by OEO to Battelle 
Memorial Institute and Education Turnkey Systems, Inc. 

The two programs resemble each other in principle, of 
course; the major distinction is that OEO designed one program, Dallas the 
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other — hence I feel I can discuss the Dallas program with more authority. 

First, the target group. Last May, we ran an analysis of 
underachieving high school students and selected a group whom, on the basis 
of our experience, we believed were highly susceptible to dropping out. 

By August 31, the first day of school, our predictions were proven 
unfortunately accurate: fully 50 percent did not show up. We divided the 

survivors into an experimental group of 960 and a control group of 700. 

The experimental group were all students in grades nine through twelve 
attending five Title I high schools. 

We decided the program should concentrate on three kinds 
of instruction: first, basic skills — communication and mathematics; 

second, occupational skills; and third, achievement motivation -- helping 
youngsters develop a determination to succeed. 

The characteristics of both experimental and control groups 
were as follows: they were 4.8 standard scores below the national 50th 

percentile in reading, 6.2 scores below on vocabulary, and 4.9 scores below 
on mathematical skills. Their teachers and counselors indicated that each 
seemed to lack any desire to succeed in school, or any realistic goals in 
life. 

When we set up this new shop, then, we chose to go after the 
toughest customers. Long before we picked them, however, we began thinking 
about the kind of program we would ask contractors to bid on. We started 
our planning in November 1969, with a Planning Advisory Group that comprised 
30 people — and I think it important to describe this group. 

The membership included only five employees of the school 
district: two central staff administrators, a principal, and two teachers. 

The other 25 included the president of the Classroom Teachers of Dallas, 
which is the local NEA affiliate; seven students and ex-students; one 
school board member; and the rest, residents of the target neighborhoods, 
representatives of local colleges, local businessmen, and officials in 
Dallas civic agencies. 

The cynical way to view this is that we were trying to 
minimize opposition - - and that, indeed, was one of the fringe benefits. 
Performance contracting seems by implication, at least, to impugn the 
competence of teachers, and one might expect their representatives to 
oppose it. But we are fortunate in Dallas to have NEA representatives who 
are equally alert to the interests of their members and to sound ideas for 
improving education. They agreed that performance contracting was a concept 
worth testing. Perhaps, they felt, it might be a step toward training 
teachers to aim for performance. 

Yet minimizing opposition was not our principal objective 
in expanding the membership of the Planning Advisory Group. It is difficult 
for any educator to admit that laymen might know a thing or two about 
educating, but we decided to investigate the possibility. 

Our humility paid off. Among many other benefits, it led 
us to include a somewhat offbeat course in the occupational training 
portion of our request-for-proposals : drafting for girls. That suggestion 

came from the businessmen on the advisory group — and every girl enrolled 
in the drafting program has already been spoken for by a local industry. 



By February 1970, the Planning Advisory Group had helped 
us develop a ’’wish-list”: what we hoped the contractors could do for us. 

By April, we had refined that list into the RFP. (The RFP is simply a 
request to eligible companies to submit proposals to direct the guaranteed - 
performance programs. It includes all the constraints imposed by the 
district, such as limits of students and financial resources, and J5_out- 
lines the objectives as perceived by the district. Dallas mailed RFP s to 
thirty-one companies.) We held a pre-bid conference in May, and chose the 

successful contractors in July . 

Now — what had we asked for in the RFP? 

The total list of performance criteria and conditions is 
much too exhaustive to repeat here. The most important requirements 
those which, I believe, you will be interested in — are these: 

First, in mathematics and communications, the students would 
have to gain 1.4 grade- levels in one scholastic year — in contrast to the 
0 0 5 grade- levels this particular population had been gaining. Payment to 
the contractor would be based on individual student gains; unless every 
student achieved a 1.4-year gain, the contractor would not be able to recoup 

his costs • - , , 

Second, in achievement motivation, the contractor would have 

to reduce dropout rates below those of U.S.O.E.'s five most successful 
Title VIII dropout prevention projects throughout the United States. The 
retention rates, however, would not be based on attendance in the achieve- 
ment motivation classes — since all a contractor would have to do to main- 
tain high attendance would be to make these classes fun. Rather, measure- 
ment of the effectiveness of the achievement motivation classes would be 
based on attendance in the math and communications classes. 

Third, with regard to occupational training, we could not 
define performance criteria as strictly as we could with the other two 
components. The essential test of occupational training is employability — 
but this is affected by economic conditions as well as by educational 
excellence. However, we did the best we could to specify performance 
standards for this component by enlisting 25 local companies to work with 
the contractor j they participated not only in the actual training, but also 

in judging the quality of the program. 

The New Century Company, a subsidiary of the Meredith 

Corporation, won the contracts for communications and math. Thiokol 
Corporation won the contracts for achievement motivation and occupational 
training. We also requested proposals for two other components: audit 

and management support. . . 

Audit, essentially, is intended to keep everybody honest — 

to prevent a repetition of the unfortunate experience in Texarkana. We 
wanted an outside agency to approve the tests given to experimental and 
control group students throughout the program; to check our research design 
so that we could appraise the effectiveness of various treatments, singly 
and in combination; to ascertain the reliability of data; and, finally, to 
certify the results so that the contractors could be properly compensated. 

In contrast to the instructional components, which were 
contracted for on a penalty- incentive basis, the audit contract was for a 
fixed fee. We chose Educational Testing Service to provide the audit. 



Management support, as the name implies, is to help out 
management — in this case, the Dallas school system* Performance contract- 
ing was new to our staff; all of them have full-time duties, and we did not 
want to divert them to an unfamiliar job. Hence we contracted with the 
Council of Great City Schools — again for a fixed fee -- to provide a 
supplemental staff that would act as liaison between the school system, the 
contractors, and the auditor. 

In addition, the Council of Great City Schools felt that 
placing a few of its representatives on our staff temporarily would 
increase their expertise in performance contracting. In a sense, even 
though their people have significant experience in this area, they would be 
serving an internship — learning along with us so that they could later- 
help other school systems. 

Hie last aspect of performance contracting that I feel you 
should know about is the "turnkey" aspect. The three instructional compon- 
ents of our program — math, communications, occupational training — employed 
the contractors' methods and materials, but they employed Dallas personnel. 

We insisted on this in our RFP. Moreover, we insisted that the contractors' 
programs be so designed that they could be adopted throughout the school 
system if we elected to do so. 

That is what "turnkey" means. Thus performance contracting 
can be viewed not only as a tool for improving student achievement, but as 
a tool for improving the effectiveness of teachers. Each contractor agreed 
to train our teachers in his methods if those methods work. Each also 
agreed to supply us with his real expense figures, so that we could appraise 
the cost-effectiveness of his program. We expected each of them to make 
a profit; we signed the contracts, and if they could deliver, we didn't 
care how much each of them made. But we did want to be able to compare 
their learning-unit costs against ours, so we could decide whether their 
methods could be extended to other students within our budget restrictions. 

In connection with the "turnkey" aspect of the program, I 
venture the opinion that performance contracting poses no threat to any 
school district's teachers. But it does pose a threat to teacher-training 
institutions. If Thiokol or New Century or Jim-Dandy Educational Systems 
can teach teachers to teach potential dropouts to read, after all the 
tenured Ph.D.'s in our universities have so resoundingly failed — then I 
predict, we will see a lot of Ph.D.'s out of work during the next decade. 

It is too early in our experiment to judge all the results. 

We do know that our target population had a much higher attendance record 
than their controls; these youngsters whom we identified as probable dropouts 
-- probable , not possible -- showed up much more often than they had in 
previous years. 

The evaluation of the 0E0 project in elementary and junior 
high reading and mathematics has not been made available to the participating 
school districts at this date. Indications are, however -- based on the 
results of interim performance objective tests — that students averaged more 
than one year's growth during one year of instruction. These results suggest 
that individualized systems of instruction below the high school level, when 
supported by sound, almost continuous monitoring and staff development, stand 
a good chance of succeeding with educationally disadvantaged students. It is 
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difficult at this point to generalize about /'individualized systems, but 
most observers agree that the teachers in the QED program generated much 
enthusiasm among their students, bringing about an obvious zest for learning 
among the participants. Some studies, though, have discovered negative 
correlations among disadvantaged students /between 8C0 * e ® ln f* 1 *”®*. 
referenced tests and scores in standardized tests -- indicating that the 
cultural bias of standardized tests might be influencing results, that the 
criterion referenced test items are too, easy, or that any number of additional 

factors could be blamed for the disparate results. 

The high school reading and mathematics program under New 
Century did not bring about the expected results , although a cursory examina- 
tion indicates that the New Century students did make greater gains than did 
the control group, even though the company did not approach its expected 
gains. An economist might consider the results the best of all worlds. The 
company did not receive its total fee because it did not reach its guarantees; 
at the same time, the evaluation did indicate a slight, if not statistically 

significant, supremacy for the treatment group. 

The initial design for the high school reading and mathematics 

program called for all students from a 1400-student target population to be 
randomly assigned to experimental and control groups. As was mentioned 
earlier, many of these students did not show up for school during the first 
three or four weeks; consequently, random assignment was virtuously impossib e. 
As the less interested students began arriving, they were put into a treat- 
ment group, leaving a great disparity in aptitude between the treatment and 
control groups, even though both groups were from the target population. 

The bias resulting from this disparity was co-varied out in all analyses -- 
an almost totally unsatisfactory technique because of the difficulty of totally 
adjusting for mathematical differences in measured intelligence. The average 
intelligence of the treatment group for mathematics was 75.16, while the 
average for the control group was 86.31. The average intelligence of the 
treatment group for reading was 72.39, while the average for the control 
group was 82«16# These figures indicate that although the research design 
called for students with IQ's of less than 70 to be excluded from the 
program, principals let many of these special education students enter the 
treatment groups while excluding them from the control groups. Some of our 
principals needed desperately a place to put their problem students. Inasmuch 
as many of them are more interested in kids than they are in research (this 
is not all bad), they decided the New Century program with all its carrels, 
hardware, and contingency management systems was a likely choice. 

Students in the mathematics program averaged .33 years mean 
gain based on the results of pre-and-post standardized tests. The control 
group, although averaging 11 points in IQ more than the treatment group, 
lost .09 years. Although the gain was greater for the treatment group, it 
is not great enough to be deemed significant at the .05 level of confidence; 
therefore, the conclusion was that performance among groups did not differ 
more than what would be expected by chance. But again the results were 
confounded by the lack of randomness within the two groups. 

That the correlation between the scores on the criterion 
referenced tests and the post standardized tests was a mere .06, indicating 
no relationship between scores on the two types of tests, was an interesting 
discovery. Other major findings of the mathematics program are: (1) students 

with higher IQ's performed better; (2) eleventh and twelfth graders performed 
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better than ninth and tenth graders; (3) the more a student was absent, the 
poorer he did in the program. These three conclusions are not very startling, 
but they do provide corroboration for some of our biases. 

Students in the reading program gained an average of .53 years, 
while their counterparts in the control group gained .48 years. As with the 
mathematics students, this conclusion persists in spite of the fact that the 
average IQ of the students in the treatment group was 10 points less than 
the average of the students in the control group. But again, the differences 
in achievement were not greater than would be expected by chance. 

The finding of no significant difference is difficult to 
explain. 315 of the 334 participants in the reading program scored better 
than 75% on their criterion referenced tests. The correlation between scores 
on the criterion referenced tests and scores on the post standardized tests 
was -.15 — not a very significant correlation, but one which indicates that 
students who tended to do better on the criterion referenced test tended to 
do poorer on the standardized test. 

In addition to the questions concerning difficulty of criterion 
items and cultural biases of standardized items, perhaps the most relevant 
question we could ask would be "how do we get disadvantaged students to do 
their best on post tests?" One student whose scores indicated an eight year 
loss in reading ability -- a preposterous conclusion -- had actually answered 
more than 75% of his criterion test items correctly. Three more whose scores 
indicated a loss of six years also answered more than 75%. of the criterion 
test items correctly. In fact, 79 students showing a loss of 1.0 years or 
more in reading fell into the same category of 75% or better on the interim 
performance objective tests. As in the mathematics program, the brighter 
students in reading performed better; the eleventh and twelfth graders 
performed better than the ninth and tenth graders; and the more the students 
were absent, the poorer they did in the program. One additional finding 
relates to the reading portion of the experiment only: females performed 

better than males. 

Reading and mathematics scores were increased if the students 
were also assigned to achievement motivation classes. Attendance in these 
classes, a segment of the contracted program with Thiokol Chemical Corporation, 
definitely contributed to greater reading achievement. Students in achieve- 
ment motivation classes attended school — their regular classes, not just 
the achievement motivation classes — 87% of the time, a marked improvement 
over the 73% attendance of the target population the previous year. Results 
of deportment scale evaluations of regular teachers would indicate that the 
attitudes of these students toward school was superior to the control stu- 
dents, even though the measured aptitudes of the control students were higher 
than those of treatment students. 

Vocational training under Thiokol was in three areas: auto 

mechanics, machine metals, and drafting. 30% of all students in the three 
courses reached the graduate level of training and were placed onto jobs. 

17% reached the apprentice level. 23% achieved the level of assistant, and 
25% reached the helper level. Since all of these achievement levels are 
varying levels of employment, 95% of the students enrolled into these 
specialized, individualized programs achieved an exit level of employment. 
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Inasmuch as all reading and mathematics participants did 
not make positive gains, the computation of costs per learning unit was 
confounded. This computation, which was the responsibility of the manage- 
ment support group, was made by assuming that all students making a negative 
gain made zero gain. In reading, of those students for which both * t *~ aaA l 
post test scores were available, 213 made some kind of gain and 181 i no 
Using 1.0 as the base performance unit, it would require $374.00 invested 
per student for each one year growth in the program. 

In mathematics , 188 of 319 students made some gain, while 
131 made no gain at all. The cost of this program was $442.38 per student/ 
year growth. Of course both of these figures would decrease if more students 
could be moved from the negative gain column to the positive side. 

We should be discouraged in Dallas inasmuch as our high school 
reading and mathematics programs did not approach our expectations, but we 
are not. This year is the first time we were able to tell our citizens, 

"Here is where X amount of your dollars went, and here is the amount of 

difference which that investment produced." . ,, , 

We know that a large inner city is a difficult place in which 

to carry out controlled experimentation, but we are not quitting. We are 
encouraged somewhat by the results of the interim performance bests. We 
are encouraged by the interest displayed by primary and junior high 
students in our 0E0 project. We are encouraged by the effect the achievement 
motivation classes had on attendance and achievement. We are very encouraged 
by the performance of our vocational students. 95^ of them did in one year 

what it has taken two years in the past to accomplish. 

Besides, we were not really evaluating performance contracting. 
Rather, we were evaluating certain instructional systems, and some of them 
will always work better than others. Our direction has changed somewhat for 
the coming year, and hopefully we will not repeat the same mistakes. We 
will be concentrating on reading in the early grades as over 11,000 primary 
students begin operating tinder some kind of performance contract during the 
1971-72 school year. 

We believe the results of our first year experience offer some 
hope. We have not found nearly enough answers to the learning problems of 
the disadvantaged child; but if one thing doesn't work, we will try some- 
thing else. We feel we owe this to the citizens who are investing their 
taxes in the special knowledge which professional educators claim to possess. 
More important, we owe this to the parents who are investing their children 

in that special knowledge. ^ 

Most important of all, we owe it to the students, for they 

are investing themselves . Whether they know it or not, those children whom 
we term the " culturally disadvantaged" place most of their hopes for the 

future on the line when they enter our schools. 

Accountability asks educators to place their careers on the 
line. Since our students had no choice of schools, but we had a choice of 
careers, this seems tome a fair trade. 
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TEACHING PERFORMANCE TESTS 
AT THREE LEVELS OF ACCOUNTABILITY 



W. James Popham 



(W. James Popham is Professor at the Graduate School 
of Education , University of California , Los Angeles* 
He received his A*B* degree from the University of 
Portland, as well as his M.A. also, and his Doctorate 
degree from Indiana University* ) 



Most current conceptions of educational accountability recog- 
nize, in one form or another, the educator's responsibility to promote 
worthwhile changes in the skills, knowledge, and attitudes of learners. 
And in recent months there has been ample rhetoric persuading educa- 
tors that an accountable stance is, indeed, meritorious* Yet, talk 
is notoriously inexpensive. So, until we devise some practical pro- 
cedures for promoting workable systems of educational accountability, 
this phenomenon, as so many other educational breakthroughs, will 
remain an improvement vehicle at only the verbal level* 

In the following remarks an effort will be made to supply more 
tangible suggestions for implementing accountability systems by first 
describing the teaching performance test, a specific measurement 
tactic which can be employed in various accountability approaches, 
next by distinguishing between three different forms of educational 
accountability and, finally, by illustrating how teaching performance 
tests can be employed in each of these three accountability systems* 



A 




The Teaching Performance Test 

Perhaps the most perplexing problem facing educational re- 
searchers and evaluators since the turn of the century has been how 
to measure a teacher's instructional skill* The most widely used 
measures, i.e. , ratings, classroom observations, and pupil performance 
on standardized tests, have all proved dismally inadequate.* There 
were two major defects in their approaches. First, they were often 
too process-focused} they tried to isolate M good teaching techniques" 
even though subsequent research strongly suggests there are few, if 
any, pedagogical ploys which will invariable work in the myriad in- 
structional settings teachers encounter. Second, if not process- 



*A more extensive analysis of the defects in such measures, and the 
advantages of the teaching performance test approach, can be found in 
Popham, W.J *, "Performance Tests of Teaching Proficiency! Rationale, 
Development, and Validation." Anericaa Educational Research JffWfliaA, 
volume 8, number 1, January 1971. 
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focused these measurement techniques failed to take into account the 
fact that different teachers pursue markedly different goals and, 
therefore, comparisons of teachers with differing instructional enr*. 
phases "yielded little more than confusion. 

To eliminate some of these difficulties, a previously untried 
assessment technique has been experimentally tested since 1965, namely, 
the teaching performance test. While there is still considerable 
research work to be carried out regarding various aspects of teach- 
ing performance tests, they have been employed in a sufficient number 
of limited field trials with results which appear to warrant their 
further utilization. 

First, what is a teaching performance x test? 'Such tests provide 
an estimate of a teacher* s ability to produce a prespecified behavior 
change in a group of appropriate learners. Here*s how it works. 

First, a teacher is given an explicit instructional 
objective along with sample measurement items showing 
how the objective* s achievement will be measured. 

Second, the teacher is given time to plan a lesson 
designed tc achieve the objective. 

Third, the teacher instructs a group of pupils for 
a specified period of time, perhaps as few as a half 
dozen students or as many as a whole class. 

Fourth, the pupils are measured with a posttest based 
on the objective but unseen previously by the teacher. 

Pupil attitudes toward the instruction are also 
measured. These measures of pupil cognitive and 
affective results serve as an index of the teacher *s 
effectiveness. 

Performance tests have been used successfully for periods as 
short as 15 minutes or as long as ten hours. One of the questions 
currently under research analysis is how long must the instructional 
period be to provide a reliable estimate of the teacher *s instructional 
skill. Similarly, there is interest in many of the shorter, more 
readily usable, performance t ests employed to accurately predict a 
teacher's success in longer, more realistic teaching situations. 

In general, the subject matter employed for such performance 
tests is novel, thereby reducing the likelihood of the learner's pre- 
vious familiarity with the topic. Becuase the. same instructional 
objective is employed for all teachers completing a given performance 
test, it is legitimate to compare different teachers with respect to 
their skill in accomplishing such preset objectives. This, of course, 
is the new neasurement angle. By holding constant the instructional 
task, it is possible to contrast the ability of different teachers 
to master the task. 
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The trick, clearly, is to control other relevant conditions so 
that all teachers have the same opportunity to display whatever in- 
structional skill they possess. This means randomly assigning learners 
to teaching and, in some situations, statistically compensating for in- 
equities in disparate learners* entry behavior. 

As indicated before, there are still important procedural 
problems, that must be solved, but even now the teaching p ®**° r “* nce 
test can play two valuable and practical roles in relationship to 
^ScaUonSl accountability systems. First, teaching performance t«ts 
can be used for instructional improvement, that is, to help teachers 
get better at promoting beneficial changes in learners. Second, 
teaching performance tests can be employed fo* skill assessment, that 
is, to discover which teachers are particularly good or particularly 
bad at this type of instructional task. In later pa * a 9 ra ^% s ° m ®. 
examples will be given of how performance tests can be used for both 
of these purposes under various accountability systems. 

Before leaving teaching performance tests, however, a few 
points must be made to assuage the doubts which generally arise when 
educators consider this measurement approach for the first time. 

Beyond the procedural questions previously mentioned, there are typ- 
ically concerns regarding the basic validity of this strategy. Note 
that teaching performance tests assess only one competency of a 
teacher, namlly, his ability to achieve pre-specified objectives. 

There are surely other dimensions which should be xn Y°*J'®<V in 
ting a teacher. Performance tests yield only one criterion. But it 
is an important criterion. Insofar as one believes the ® x ssion of 
teachers^is to change children for the better, then any indicator of 
a teacher »s skill in doing so should be given careful consideration. 

Some will say, "But this isn*t what teaching is really like. 
Teaching for a full day with 35 kids in a classroom is vastly diff- 
erent than teaching eight randomly assigned learners for a 30 minute 
lesson.** Of coursl there are differences. But is there any reason 
to believe that a teacher who has performed miserably on several short 
term performance tests will suddenly blossom with pedagogical splendor 
in a regular teaching situation? Any kind of prediction involving 
human behavior is made probabilistically. Teaching performance tests, 
even in their current unsophisticated form, permit us to sharpen our 
predictions. 

Educational Acco untability Systems 

tyoyr turning to a consideration of educational accountability- 
systems, let's examine the key ingredient in such approaches, for 
although we shall examine three different methods of implementing 
systems of educational accountability, there is a basic similarity in 
all of them, namely, an assumption that those designing or implement- 
ing educational systems must become responsible for the 
systems produce in the intended learners. This re JP®" s ^ U ^ t f 8 no J t 
discharged merely by asserting that one is responsible but, rather, it 
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requires the accountability or led ted educator to produce evidence 
■ regarding the out cones that have been produced in learners as a 
consequence o f his instructions* 

This evidence is then made available to different individuals, 
since the accountable educator stands ready to accept positive or 
negative judgments of his instructional efforts and the subsequent 
actions which nay stem fron those judgments* Indeed, the fact that 
evidence regarding the results of instruction is demanded by differed 
groups or individuals permits one to distinguish between alternative 
systems of educational accountability* /There are major differences 
in the kinds of decisions which are made regarding an instructor's 
efforts depending upon who is making the decision* We can consider, 
thfirafoze, three rather distinctive forms of educational accountabi 1> 
ity, persona 1 accountabi lity » profe ssional accountability, and public 
accountabi lit v* » 

The chief difference in each of these three forms of accounts* 
bility is the nature of the decision maker who demands evidence in 
regard to the outcomes of instruction* In the case of personal ac- 
countability, <an instructor is. the initiator of any review of the 
results of his own instruction* Others may be called in, for instanq 
one or more colleagues to participate in a review, but it is the in- 
dividual teacher who decides whether the review of his instructional 
results will be solitary or include others* For example, a teacher 
who carefully evaluates the kinds of changes in learners yielded by 
his instructional efforts, then makes decisions regarding the modifi- 
cation of his teaching tactics, would be engaging in a form of personal 
educational accountability* NO one else need be involved in such 
decisions, but any decisions regarding instructional modifications 
are not capricious* Rather, the educational practitioner who is pers- 
onally accountable is engaged in a systematic accum ulation of evideice 
regarding the good or bad results of his teaching* Moreover, he is 
prepared to take the action dictated by the evidence. If he consults 
others regarding these activities, it is a totally individual decisis. 

Professional accountability, on the other hand is initiated by 
a group of the instructor's colleagues, such as the faculty members of 
his school or school district, or perhaps his teachers' organization. 
The choice to review the results of instruction is not the individual 
teacher's to make. There is an element of imposition present in this 
form of accountability, but the imposed demand for accountability 
emerges from within the teaching profession, rather than from external 
quarters* suppose, for example, that a high school faculty voted over* 
whelmingly to set up. a. system whereby each teacher would have to pro- 
duce tangible evidence of the kinds of learner progress which was 



*A more detailed treatment of these accountability approaches is 
available in a filmstrip-tape program from Vimcet Associates* Alterna- 
tive Avenues to Educational Accountabi lity . P*0. Box 24714, Los 
Angeles, California 90024 
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being made each month , such evidence to be reviewed by specially des- 
ignated teachers representing the subject field involved* The results 
would be made available only among the faculty, with the primary pur- 
pose being to identify ineffective instructors so they could be helped 
to improve* This would be an instance of professional educational 
accountability • 

Public accountability* as might be in / erred from its name, 
occurs when evidence regarding the quality of learner attainments is 
demanded by the public* For instance, if the school board requires 
that reasonably interpretable evidence of learner achievement be sup- 
plied to school district taxpayers each year so that laymen can reach 
results-based judgments regarding the school system's effectiveness, 
this would be a clear instance of public educational accountability* 

In examining these three avenues to educational accountability, 
we see that the common dimension in each is a commitment on the part 
of the educator to assume responsibility for the results of instruc- 
tion* Evidence regarding the impact of instruction on learners is 
systematically accumulated and then made available to one or more aud- 
iences* If the initiator of the evidence-appraisal scheme is oneself, 
then personal accountability is involved; if the initiators are one's 
colleagues, then professional accountability is present; if the init- 
iators are lay citizens or their representatives, then public account- 
ability is involved. 

Now each of these three forms of educational accountability 
carries with It the potential to improve the quality of education* 
However, each of the three forms contains some inherent strengths and 
weaknesses which we should recognize* Those who mindlessly toss these 
three variations into the same accountability bag really confuse the 
picture. Accordingly, we shall examine some pros and cons of each of 
the three alternatives* 

Turning first to personal accountability, how might it work? 
Well, clearly the focus in this approach is self-improvement. A 
teacher wants to get better at what he's doing and becomes personally 
liable for the changes he is producing, or not producing, in his 
students* Thus, for example, we might find a teacher who prior to an 
instructional sequence routinely establishes expected levels of leans 
proficiency* By comparing the:. actual levels of learner progress after 
instruction with the hoped-for progress, the teacher can reach a more 
defensible judgment regarding his instructional effectiveness. It^ 
must be stressed that a personal system of educational accountability 
involves more than a casual ,, look-see*' at the close of an instructional 
sequence, even if other teachers participate in the "look". Most 
teachers typically get a somewhat ill-defined impression of whether 
they have been teaching well* A true system of personal educational 
accountability involves the establishment of a s^fstemati£ mechanxsjn 
whereby the instructor can appraise the quality of hxs efforts in 
terms of evidence regarding learner growth* 



The fact that in a personal educational accountability scheme 
the teacher is accountable to himself is at the same time the most 
prominent strength and the greatest weakness. Since the teacher is 
the exclusive initiator of the system and, if he wishes, evidence of 
pupil performance is examined by no one but the teacher, the approach 
is obviously not very threatening. As a consequence, a good many 
teachers should be willing to employ it. On the other hand, since 
personal accountability schemes are essentially volitional and often 
private, it may develop that the instructors who most need to improve 
themselves will be the least likely to engage in such approaches. For 
the weak teacher, even a private mirroring of his mediocrity is not a 
pleasant experience. . 

Now let*s turn to professional educational accountability, for 
while this less private. form of accountability fails to capitalize on 
the chief strength of a personal accountability system, it does cor- 
rect a major weakness of the personal approach to educational account" 
ability. The distinguishing feature of a professional accountability 
system is that it is initiated by a group of professionals, such as 
a teachers* organization. Assuming it has the power to implement its 
plan, if the professional group requires teacher participation in the 
enterprise, then teachers will have to participate whether they wish 
to or not • This departure from the completely volitional nature of 
personal accountability obviously makes professional accountability a 
more threatening concept for many educators. On the positive side, 
the possibility that incompetent teachers could avoid participating in 
the system is eliminated by requiring their involvement. 

In examining the merits of professional accountability we see 
immediately that since it is an intraprofessional operation it would 
be less threatening to many teachers, even if required of all. This 
is its major strength and represents the reason why a number of educ- 
ators believe professional organizations may lead the way in promoting 
educational accountability systems. On the negative side, it must be 
recognized that professional associations have historically been very 
reluctant to expose or expell their ineffective members. Thus, the 
possibility exists that even if the professional group is unable to 
improve certain of its members* skills, they may not be willing to 
make the hard decision to re-assign or even release a weak member. 

Turning to public accountability we find that citizens may act, 
typically through their elected representatives such as school boards 
or legislators, to require a system of educational accountability. 

For example, suppose a state legislature enacts a law requiring all 
district’s in the state to choose one of several alternative schemes 
(devised by the legislature) for annually releasing to the public.', 
evidence regarding the results of instruction.* Such a system is a 
clear instance of a public accountability scheme, the effectiveness of 
which would depend on the quality of the procedures worked out by the 
legis lature. 
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Nbw the chief advantage of a public accountability approach is 
that because it is imposed from the outside on the educator there is 
less likelihood that instances of ineffective instruction will be 
tolerated. And since all forms of educational accountability are de- 
signed to improve the quality of instruction our children receive, 
that is clearly a dividend. On the other hand, since imposed from 
without, we shall surely find that many teachers will resist public 
accountability schemes with a vengeance. Some will resist because 
they have basic doubts regarding the wisdom of such approaches. Some 
will resist because of personal fears, for surely the individual most 
terrified by a full blown accountability system is the person least 
able. • to promote demonstrable growth in learners. It is also^not 
difficult to imagine poorly conceptualized public accountability sys- 
tems operated by ill-informed or politically repressive individuals. 
Some people, fearing such misuses of a system, may resist all account- 
ability schemes. 



Using Performance Tests at Three Accoun tabi lity Levels 

Now quite obviously one can employ all sorts of measurement 
tactics to flesh out any accountability strategy, -his paper will 
only consider how teaching performance tests might prove useful. 

In the case of personal educational accountability we can see 
how a teacher might use a performance test chiefly for instructional 
improvement. Either by himself or with invited colleagues, we can^ ^ 
visualize an elementary teacher working with different groups of chil- 
dren for short periods after school in successive efforts to improve 
her skill on a particular performance test, ^>r on a certain class of 
performance tests. Lessons which failed to achieve the objective 
would be revised. If a lesson was successful, its features would be 
noted for subsequent use, Whenever local or national mormative data 
were available, a teacher might wish to contrast her performance on 
given performance tests with those of other teachers. If she finds 
that she ranks at the 70th percentile, perhaps she will be sanguine. 

If she falls at the 15th percentile, then improvement is clearly die- 
tated* To recap* in personal accountability systems the chief use of 
teaching performance tests will be directed toward instructional im- 
provement. 

For' prof essional accountability systems, however, teaching 
performance tests can be employed for both instructional improvement 
and skill assessment. Regarding instructional improvement, it is 
apparent that groups of teachers might wish to foster the use of per- 
formance tests to help their colleagues get better at accomplishing 
instructional objectives. Teachers would be required by a group of 
colleagues to participate in a series of performance test clinics which 
featured post— lesson clinical analyses of the tqacher*s instructional 

decisions. 

But more importantly, perhaps, is the possibility that teachers* 
organizations will seize upon the use of teaching performance tests as 
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a skill assessment device to accomplish what they have always sought- 
control over entry into the profession. Consistent with a general 
thrust for professional responsibility, teachers* organizations might 
S fl* Up P r f cedu f es ». t 3 'pically involving state teacher credentialling 
°*“ ces » * n which individuals aspiring "to be teachers would be 
obiiged, along with other factors, to display at least a minimum level 
or skill on teaching performance tests. 

Paced with external scrutiny and the possibility of imposed 
accountability schemes, the astute leaders of teachers* associations 
may also see the possibility of employing teaching performance tests 
!n a self -regulatory system for all of their members. These teachers 
who displayed little ability to master such teaching performance 
tests could be given special instructional assistance via a service 
provided by their own teachers* organization. Such collegial improve- 
ment schemes might be seen as a means of warding off externally 
imposed teacher appraisal activities. Thus, for the professional form 
of educational accountability we readily see uses of performance tests 
for both instructional improvement and skill assessment. 

Finally, for public accountability systems it appears that the 
skill assessment use of performance tests will predominate* In an 
effort to create school systems which can accomplish results, it is 
certainly plausible that the public might demand that teachers display 
at least minimal proficiency on teaching performance tests. 

One obvious form this skill assessment approach might take 
stems from the fact that irt many subject fields there are currently 
more applicants for teaching positions than there are jobs. Admin- 
istrators must choose among these applicants and, for the good of the 
children, .will wish to select the most competent teachers from the 
many applicants. As an aid to selecting the very best applicants, 
the school district could set up a series of teaching performance 
tests which must be completed by all district job applicants. The 
amount of time needed from applicants might be three or four hours. 

In many districts this type of applicant screening examination would 
be' similar to the procedure whereby applicants for a given graduate 
school must complete, at their personal expense, the Graduate Record 
Examination whose results are considered along with other information 
in deciding whether to admit the applicant to graduate school. And 
since enrichment topics could be used for the performance test sub- 
ject. matter, any. pupils from the district who participated in this 
applicant screening would be gaining new knowledge. 

Reflecting the fact that such schemes have now moved well 
beyond the theoretical stage, one firm. Instructi onal Appraisal 
Services*, now offers school districts a complete service for the 



*Additional information can be obtained from Instructional Appraisal 
Services, 105 Christopher Circle, Ithaca, New York 14850. 
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screening of job applicants via teaching performance tests in which 
(1) the district supplies only the physical facilities for the testing 
plus a limited number of school children for a series of short enrich- 
ment lessons, (2) the main cost of the testing is borne by the job 
applicant, and (3) the appraisal service supplies complete supervision 
of the testing program plus, at its conclusion, a ranking of teacher 
applicants according to their skill in promoting different types of 
cognitive objectives as well as positive learner affect* 

There are, of course, other ways in which teaching performance 
tests can be employed in public accountability systems* Indeed, the 
imaginative educator will find numerous uses for these tests, both in 
connection with instructional improvement and skill assessment. 

The reason that the teaching performance test strategy should 
be particularly useful to proponents of educational accountability 
systems is its complete congruence with the central assumption of all 
accountability systems - a focus on the outcomes of instruction. 
Predictably, there will be procedural problems in employing such 
performance tests to promote different forms of educational account- 
ability. But, as suggested at the outset of these remarks, it is 
only when we descend from the ethereal plateaus of rhetoric to the 
practical world of public schools that such problems can really be 
faced and solved. 
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SYMPOSIUM I 



DEVELOPING BEHAVIORAL OBJECTIVES 



Chairman: Dr. Jack M. Thompson 

Director of Instruction and Guidance 
Sonoma County Schools 

Participants: Donald Kase, Consultant 

Larkspur 

Roy Carson, Mathematics Consultant 
Sonoma County Schools 
Santa Rosa 



PROBLEMS OF IMPLEMENTING THE WRITING OF BEHAVIORAL OBJECTIVES 



V 



Roy M. Carson, Consultant 
Sonoma County Office of Education 



Much enthusiasm has been generated among administrators to imple- 
ment the writing of behavioral objectives at the teacher level, as a savior jj 

for determining whether a student has learned a specific concept. A new 
twist for sending these messages down to teachers is a coined phrase, that j 

by 1972 all teachers will be required to write behavioral objectives due to i 

the state implementing P.P.B.S." j 

Many consultants began retraining for the task to conduct inservice j 

training programs for teachers centered around writing behavioral objectives. j 

The major problems that existed in conducting these programs were as follows: j 

1. Teachers feel that this is just another burden handed down from the j 

administrators. j 

2. Another task to incorporate into an already crowded day. j 



3. As teachers, they were not involved in the decision-making process. jj 

j! 

4. If administrators knew what was going on in the classroom, they could j 

see teachers were already writing goals and objectives through their « 

daily lesson plans. I 

V 

VI 



In honoring the request of district administrators, I conducted 
several workshops on behavioral objectives with teachers. At first there 
was a degree of nonsuccess due to the lack of communication between teachers 
and a dmini strators and the level of the presentation. (Many teachers had no 
background on the definition of behavioral objectives.) Other workshops 
involved the participants through audio-visual materials and a writing 
exercise; again a degree of nonsuccess. Success did occur when teachers 
were taken at their level of sophistication of knowledge on behavioral ob- 
jectives (Mainly working from their lesson plans and patiently bringing them 
along) . 

Stress was given to observing the behavior of students and writing 
objectives based on this behavior, rather than plugging students into a 
predetermined objective unrelated to the students in their class. Through 
this process teachers agreed that trying to state objectives in terms of 
behavior is an eye-opener for planning. 
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PROGRAM PLANNING IMPACT 



Dr. George M. Kibby 

Superintendent of Schools 

San Dieguito Union High School District 

Cardiff 



Stephenson R. Parker 
Director, Instruction Center 
California Teachers Association 

J. Russell Kent 

County Superintendent of Schools 
County Government Center 
Redwood City 
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PROGRAM PLANNING IMPACT 



ACCOUNTABILITY 
( teacher ' s viewpoint ) 



Stephenson R. Parker 



Stephenson R. Parker is Consultant to the CTA 
Curriculum and Instruction Committee and the 
CTA Instruction Center, Burlingame, California. 
He received both his A.B. and Ed. D. degrees 
from the University of California at Berkeley 
and his M.A. from San Francisco State College. 



Teachers are told that schools must be accountable to the 
taxpaying public for the funds they spend, and accountability must be 
measured in terms of an improved product. In the broadest sense, then, 
accountability from the teacher's viewpoint is that teachers would be 
accountable for the performance of their students only if they share in 
the decision-making process which has a bearing on student performance. 

Teachers want to be and must be accountable. However, 
accountability is a two-way street. If teachers are to be accountable, 
they must also be provided with the educational tools, parental, adminis- 
trative and political support needed to make accountability more than just 
a catchword. If not, teachers will be forced to use their power to assure 
that they can provide the "professional" education being demanded of them. 

For example, are teachers to be held accountable and hence, 
responsible for overcrowded classrooms in the traditional sense? Is it 
their fault, or better still, should teachers be held accountable for the 
inadequacies of the curriculum, textbooks, supplies, libraries, and 
counseling services, of which they played no part in determining? Is it 
their fault that in many school districts buildings are being condemned and 
pupils forced to double up? 

Professional behavior . The notion of professional behavior 
in education is not new; it is the nearness to it that is. What is 
professionalism? Simply, a professional professes to know better than his 
clients what ails them. The professional lays claim to the exclusive right 
to practice that profession as a vocation -- the basis of license. The 
professional is expected to think objectively and inquiringly about matters 
which may be for the layman subject to an orthodoxy and sentiment which 
limits self-inflicted painful intellectual exploration. 



The professional roust demonstrate these four attributes: 

(1) a high degree of generalized and systematic knowledge, (2) primary 
orientation to community interest rather than to individualized self- 
interest, (3) a high degree of self-controlled behavior through codes of 
ethics, internalized in the process of work socialization and through 
voluntary association organized and operated by the work specialists them- 
selves, and (4) a system of rewards (monetary and honorary) that is 
primarily a set of symbols of work achievement, and thus ends in themselves 
as opposed to some end of individual self-interest. 

These four essential attributes (the sociology of a profes- 
sional accountability) define a scale of professionalism, and a way of 
measuring the extent of accountability. 
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THE IMPACT OF PROGRAM PLANNING. 



J. Russell Kent 



(J. Russell Kent is San Mateo County Superintendent 
of Schools and a member of the California Commission 
on School District Budgeting and Accounting . He 
received his B.A., M.A. and Doctorate degrees 
from Stanford University.) 



Do we really want to plan ? Do we really want to clarify 
problems and their full implications? Do we really want to do this in 
the full glare of the public arena? It has always been assumed that 
planning was good, and in public institutions that there should be, at 
the least, full public knowledge of the plans, if not wide public par- 
ticipation in their formulation. Do we really believe this? 

What are the hazards in public anticipation? What, for 
example, might have happened if the people of California could have 
known in 1950 that in the decade to follow California's public school 
enrollment would more than double, requiring a trebling of the invest- 
ment in school facilities, while at the same time the operating cost 
per child would more than double ? Would foreknowledge have led to 
more effective response than that which we actually achieved? Or 
would the public have backed away? 

In part, the answer to these troublesome questions lies in 
our answer to two other questions: 

(1) How extensive is our ability to foresee, and therefore, 
plan, not only for specific needs and problems, but also in the complex 
and dynamic milieu in which these problems or needs occur? 

(2) Assuming some ability to foresee with accuracy, can we 
communicate effectively to the public the highly complex nature of the 
interrelationships which are involved, in contrast to simplistic over- 
simplifications ? 



It is in the seeking of affirmative answers to these questions 
that organized planning systems such as PPBS, or Planning, Programming, 
Budgeting Systems, have begun to emerge. It is important to keep in mind 
that at this point in time such systems rest upon the hypothesis that the 
answers to the questions are in the affirmative — a hypothesis yet to be 
confirmed . 



Systems represent an orderly or sequential approach to the 
accomplishment of tasks, or to the description of events, which involve 
multiple and interrelated components. Whether devised deliberately or 
evolved empirically a system is directive, insofar as it presents a 
structure for activity. Since a certain rigidity is an inherent quality 
of any "system" , it becomes critically important to examine fully any 
system which is to operate in the area of human affairs or in human in- 
stitutions . 



A critical examination of program planning systems , as they 
might apply in California's schools leads me to the following observations: 

(1) We do not yet have a working model of a PPB system which 
may operate in a school system. We have a concept, and we have ex- 
perimental development reaching towards its realization. 

(2) The techniques of systems analysis from which PPBS dev- 
eloped work much better in some situations than in others. They work 
best where there is a single goal, where this goal is tangible and subject 
to precise measurement, and when the outcome is proximate in time, i.e. , 
when there is a minimum of time lapse between the input of effort and the 
output of product. 

(3) The nature of public schools which must deal with public 
policy and operate in the realm of public affairs characteristically is 
opposite to the conditions cited above. School goals are multiple , not 
singular, presenting a strong probability of internal inconsistency which 

is unresolved in the goal structure . School goals tend to be controversial 
rather than accepted and understood. 

School goals tend to be intangible rather than tangible. They 
are never physical products which can be weighed and counted. School 
goals are realizable in a remote rather than a proximate time period. 
Primarily we look forward to future performance of pupils rather than to 
occurrences which are to take place immediately. 

All difficulties notwithstanding, change is the hallmark of our 
age. Are we to participate in the management of change, or are we merely 
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to respond to its vicissitudes? Is there any real alternative to planning 
as a way of life in our times ? And if we are to plan in an open society 
how else can we proceed but publicly, and how else are we to deal with 
the highly complex matrix of variables but to proceed systematically? 
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SYMPOSIUM III 



EFFECTS OF ACCOUNTABILITY ON EVALUATION AND RESEARCH 



Chairman: Dr. Joseph T. Hanson 

Professor of Education 
Ambassador College 
Pasadena 

Participants: Harry Small eriburg, Director 

Division of Research and Pupil 
Personnel Services 

Los Angeles County Superintendent of Schools 

William H. Stegeman 
Assistant Superintendent 
Curriculum Services Division 
San Diego City Schools 



EFFECTS OF ACCOLlblTAblLlTV 014 EVALUATION AM RESEARCH 



Harry i-J. Smallenburg 



( Harry W. Smallenburg is Director of the. Division cf Res earch 
and Pupil Personnel Services in the Office o f the. lot AngeCea 
County Superintendent o f School A. He received his 8. A. degree 
from the University of Buffalo, and both his M.A. and Ed.D. 
degrees from the University o ( j Southern California) 



In his keynote address to the 1970 conference of the 
tdational Association of Pupil Personnel Administrators in 
Indianapolis, Vr. Leon Lessinger, former Associate United States 
Commissioner of Education, made the following statements : 

"Accountability has to do with honoring promises. It is the 
matching of intent to results; the comparison of what was supposed 
to happen to what actually happened. In education, accountability 
is the policy of demanding regular independent reports of promised 
student accomplishment for dollars provided. It is the hair-shirt 
policy — the responses to budget-passing time to the request for 
more money with the question, "What did you do with that other 
money?" It is not performance contracting , or behavioral objectives , 
or PPBS, though these inventions may be useful in implementing an 
accountability policy. In the final analysis , accountability is the 
final analysis — the hearing to get the facts, to determine worth, to 
check results. It brings to school instruction the same flavor of 
inspection and feedback brought by the fiscal auditor to school 
finance. " 



This concern fox accountability has been manifested 
in many programs and projects in California; notably, (PACE) Projects 
to Advance Creativity in Education, (PEP) Preparation of Educational 
Planners, the Program Planning, budgeting Systems, and Performance 
Contracting. The most recent illustration of this concern for 
systematic evaluation of services is the publication of the. 
California State department Oa Education entitled ” Accountability in 
Pupil Personnel Services: A Process Guide for the development of 

Objectives . " 
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Fundamental to all of these appAo aches to accountability 
oaz the. following consldeAations listed In the. above publication: 

1 ) Stated Outcome A 

2 ) CAlteJilon HeasuAes 

PAetest WeasuAes 
En Route M easuAcs 
Posttest ideas uacs 

3) Components of the Guidance PAocess 

PAocess Variables 
Situational VaAlabZeA 
Pupli Variables 
Human Support VanlablcA 

4) Cost Effectiveness 

5) Between- PAogAam Comparisons 

There Ia a gAoivlng body of evidence that these cJutenla 
of accountability oac being met In a esearch and evaluation studies 
conducted In dlstnlct and county offices as Illustrated by AepoAts 
&Aom Covina- Valley, Culver City , the Office of the Los Angeles County 
Supenlntendent of Schools, and others . 

Fas ten. and moAe satisfying progress can be made as 
dlstAlcts and county offices resolve such continuing adminlstiatlve 
problems as assignment of time and staff to AeseaAcJi activities and 
pAovlslon of InseAvlce education foA adtnlnlstrative, s upeAvls oalj , and 
teaching staff , 



Va. J. Wayne Wrightstone emphasizes that long- tern 
pAoblems relating to evaluation continue to confAont AeseaAchenA In 
evaluating pupil groioth. These Include : 

1 ) definition of noAmal gAoioth 

2 ) Reliance on IntcApolated noAms 

3) InteAlevel and hiterform equivalence 

4) Reliability of dlffeAence scoAes 

5 ) RegAesslon tomAd hie mean 

Recognizing these pAoblems of admlnlstAatlon and of 
evaluation and measurement, Aes eaAch workers can continue to ImpAove 
the quality of Aes eaAch by obseAvlng the following Aecommendations 
made to pupil personnel adminlstAatoAS by Va . LesslngeA: 

J) Help thelA personnel diagnose and describe the degAee of pre- 
dlctable tnprovement that can be achieved by each child served. 

2) Compile and audit data based on actual expedience to pAovlde 
a stoAehouse of good pAactlce. 

3) design pAeclse, Individual systems capable of Identifying, In 
tenms of peAfoAmance cArterla, the stAengtks, weaknesses and actual 
benefits obtained by each student as he pAoceeds thAough his foAmal 
education caAeeA and beyond . 

4) design pAognams to twin the staff In the effective Intro- 
duction, use and monitoring of good pAactlce . 
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5 ) Provide plans {,or involving and inclining othefi. education 
workers and the community about successful practices. 

6) Apply the recognition that all school personnel share respon- 
sibility with the home and the client for achieving results, each 
partner being accountable for executing those phases in which he is 
most competent . 



ACCOUNTABILITY AND THE FAILURE SYNDROME 



by 

William H. Stegeman 
Assistant Superintendent 
San Diego City Schools 



Public and professional demands for accountability have resulted in a 
flurry of "new programs." The first evaluations of these programs are now 
appearing. The first round with accountability shows we are failing. Is 
it possible that, inspite of publicity to the contrary, we are still trying 
to offer the same educational programs that brought about the demand for 
accountability? Are recent evaluations just a better job of revealing our 
failures? 

The widely heralded "new" reading programs gave great hope to 
disadvantaged youth and parents. "At last," they thought, "we will now 
learn to read." The results are showing the same dismal results as before. 
The truth is, new programs have continued to teach reading in the same 
institutionalized way as before. New goals, new objectives, new techniques, 
new media, new learning styles, new programs have been neglected. Instead 
we have ignored research, used the same old goals, disguised our true 
objectives, over publicized our sketchy plans, faked action, and then are 
surprised by the results of the evaluation. 

The diversity of society and the adaptive needs of students require a 
fresh look at what being educated means. If we observe the "cold war" 
decades of the 50* s and 60* s we see some disturbing trends. For example, 
the trend toward uniformity in curricula, overemphasis on academic 
credentials, growing bureaucracies as schools become larger, isolation of 
students and faculty from the "real world," and many other signs of rigidity 
make educational institutions less and less a reflection of reality. 

We can not let these trends continue* We must recreate a more diverse 
and responsive educational system. We must expand our concepts of what is 
the role of the student, what is school, what is learning, and what is a 
teacher. We must develop many alternative paths to learning. In other 
words, we must become accountable for what education accomplishes and for 
what it 8 true purpose is. 

Evaluation must be less concerned with the needs of our educational 
institution and more concerned with learning as seen by the student and 
society. The widespread assumption that the responsibility of the 
educational system is to provide opportunities for the conformingly 
successful students must be replaced. The new system tiust provide an 
exciting and useful education for every student at every step of the 
educational process. The failure syndrome revealed by evaluation 
statistics is due to the vast range of potentially productive students 



who remain outside the traditional and artificial success limits of the 
present system* The outsiders are producing a negatively weighted 
evaluation profile. 

There are some guidelines that may be useful as we reconstruct our 
objectives, and reorient our evaluation. 

... There has been reform, and its pace has been accelerated by the 
advent of student protest and demands of minority groups. 

... There are now wholly new technologies which make a new structure 
possible. 

... There is an increasing amount of research available on the 
learner and society. 

... There is a mounting conviction on the part of staff, students, 
and society that a new institution, more relevant to our needs, 
is desirable and it is the process rather than the institution 
itself that is of greatest importance. 

Along with accountability and evaluation has emerged the concept of 
"cost effectiveness." Unfortunately far too often the discussion }f cost 
is limited to the question of expenditure reduction. Cost effectiveness 
must be less concerned with lines of a budget and more with the time, 
talent and results of staff and students. A conventional budget fails to 
reflect time and talent is used efficiently. There are also subtle and 
technical problems related to cost effectiveness. It is within individual 
programs that cost effectiveness thinking may be most rewarding. This is 
where making progress less costly for students is the pay off. Even here 
we must ask whether the teacher's lecturing actually produces as much 
learning as the amount of time spent guiding independent reading, or whether 
seme type of group activity helps students grasp better the significance 
of the knowledge acquired. And finally, we must ask whether both costs and 
the quality of learning can be improved by changes in program, procedures, 
organization or content. 
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SYMPOSIUM IV 



LEGISLATIVE DEMAND FOR ACCOUNTABILITY 



Chairman: 


Dr. Thomas A. Shellhammer 
Deputy Superintendent for Programs 
State Department of Education 


Participants : 


Clinton Jordan 

Principal Administrative Analyst 
Legislative Budget Committee 
Sacramento 

Harvey Hunt 

Legislative Coordinator 
State Department of Education 



ACCOUNTABILITY OR LEADERSHIP? 



Clinton M. Jordan 



(Clinton M. Jordan is Principal Administrative 
Analyst with the Joint Legislative Budget Com- 
mittee. He received his AB and MBA degrees 
from Stanford University.) 



The term accountability has ah array of definitions from 
the narrowest interpretation of a strict accounting of the ways in which 
school administrators spend funds to the broadest interpretations of 
various systems of statewide testing devised to determine how effective 
educators have been in teaching children. It is therefore difficult to 
discuss accountability without first establishing who is going to be 
accountable to whom and for what. However, this is not a serious prob- 
lem if accountable activities are clearly defined. For example, state 
legislation requires the Department of Education to employ cost effec- 
tiveness measures in an annual evaluation of all categorical aid edu- 
cational projects for purposes of determining which school district 
projects should be expanded, modified or replaced to produce the 
highest degree of program achievement and cost effectiveness. 

The Joint Legislative Budget Committee recently com- 
pleted an assignment to conduct an independent fiscal review and 
analysis of six major categorical aid education programs in California. 

A basic conclusion of our two year study was that the data base essen- 
tial to cost-effective evaluation of the programs under review did not 
exist, nor had plans been developed by the department to insure this 
data base for future evaluations. Thus the concept of accountability 
becomes less important than a commitment by educational leaders to 
accept the yardsticks of accountability — measurable performance 
objectives and evaluation systems. In other words, the emphasis must 
change from detailing "input" quantities to analyzing "output" measure- 
ments, i.e., the progress of the student while he is in school and how 
well he is able to take his place in society when he has completed his 
schooling. 



Yardsticks for accountability can be imposed from outside 
the educational system or by aggressive leadership from within the 
system. For example, we believe that if the Department of Education is 
to fulfill its leadership role and meet the increasing demands for 
accountability at the state level, it will be necessary for the department 
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with the aid of the educational profession to develop a comprehensive 
evaluation system along these suggested lines: 

1. Develop statewide measurable performance objectives for 
every educational program or service that receives state 
support. (Districts would be encouraged to develop ad- 
ditional measurable performance objectives based upon 
their local and often unique needs, goals and resources.) 

2. Establish an evaluation plan for each statewide measur- 
able performance objective. This plan should specify: 

a. Data to be collected, e.g., student profile data, 
comparison group data and types and timing of pre- 
and post-tests in basic skills and areas other than 

basic skills, including criteria referenced tests ! 

and diagnostic tests. j 

b. Procedures for use of data including how the data j 

will be processed, evaluated and compared on a i 

statewide basis. j 

3. Rank and compare districts annually in terms of expendi- I 

tures, salient characteristics and achievement on per- j 

formance objectives as measured by statewide tests. j 

4. Publish and disseminate annually all of the above I 

information in the form of administrative guidelines, I 

standardized evaluation report forms and statewide j 

evaluation analyses. 

5. Conduct on-site review of those districts which ranked j 

highest in the statewide evaluation and those which ] 

ranked lowest. Disseminate information on the most 1 

effective educational techniques observed and provide ! 

assistance to those districts found to be least j 

effective. j 

.r 

Many fears have been expressed by educators concerning the I 

use to be made of such an evaluation system — emphasis on the dollar ! 

rather than the student, dehumanization, overreliance on test results, j 

emphasis on short-term educational performance, oversimplification of the \ 

learning process and implications of a system of rewards and punishments j 

to name just a few. Any evaluation or management data system may be mis- 
used. Nevertheless, decisions based upon standardized data and a rational ! 

plan provide a far greater assurance that they will be the right ones than ! 

decisions made without such aids. j 

i 

Demands for accountability are increasing from those who I 

question whether the benefits of public education are commensurate with j 

escalating costs* We believe the leadership of professional educators in ' 

establishing evaluation systems is a necessary response. j 
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ACCOUNTABILITY: THE SEARCH FOR 



UNDERSTANDING IN PUBLIC EDUCATION 



Harvey K. Hunt 



(Harvey K. Hunt is Legislative Coordinator of 
the State Department of Education* He received 
his B*A* degree from Whittier College and was 
a CORO Foundation Fellow in Public Affairs*) 



There exists today a crisis in confidence in our public schools 
which if not met head-on immediately will seriously threaten the system for 
years to come* The signs of concern have been with us for some time* 

• The general resistance of the electorate to approve override and 
bond elections 

• The unwillingness at the state level to restructure our outmoded 
system of school finance 

• The variety of popular literature which criticizes and downgrades 
the operation of our schools 

• The introduction of panaceas, each of which is sold as the simple 
answer to all of our problems 

Unfortunately, there is no simple response to the critics 
which would restore confidence in the system, since (1) there are no universal 
rights and wrongs and (2) the expectations for and the role of public educa- 
tion is constantly in a state of flux* 

How then can we meet the challenges presented to us by the 
loss of confidence when we are dealing with a pluralistic system where our 
role and responsibilities are constantly changing? 

He must turn to the sources of criticism and carefully examine 
the concerns which are expressed* A Gallup Poll in 1970 found that 67 per- 
cent of the public believed that teachers and school administrators should be 
held more accountable for the progress of their students* Webster indicates 
that if you are accountable you are liable to be called upon to account or 
answerable and that you are able to -account for or explain what you are doing 
at any time through the use of reliable indicators* 

Consequently then to increase the accountability of the system 
is to increase public understanding of the facts indicating the strengths and 
weaknesses of our educational process* In effect this means to completely 
open the schools to scrutiny and criticism and to be willing to accept re- 
sponsible suggestions when made* There is a substantial danger in this 
approach which most professional educators would be quick to point out* If 



we simply throw open our books to public view the untrained eye may be lead 
to all sorts of erroneous conclusions. The problem is aptly summarized by 
Dewey in The Public and its Problem when he states that 

"if one wishes to realize the distance which may lie between 
'facts and the meaning of facts,' let one go to the field of 
social discussion. Many persons seem to suppose that facts carry 
their meaning along with themselves on their face. Accumulate 
enough of them and their interpretation stares out at you. The 
development of physical science is thought to confirm the idea. 

But the power of physical facts to coerce belief does not reside 
in the bare phenomena. It proceeds from method, from the tech- 
nique of research and calculation. No one is ever forced by 
just the collection of facts to accept a particular theory of 
their meaning so long as one retains intac t some other doctrine 
by which he can marshal them . " (Emphasis added.) 

The problem of accountability that we are faced with today is 
not the result of the absence of facts and data for the public to view on the 
subject—it is the overabundance of information. What is really required to 
be accountable is not the production of miles of computer printouts or re- 
search papers that gather dust, but the leadership of educators in communi- 
cating to the public in layman's terms what all of this information means in 
terms of the education of their children— where we are doing a good job and 
where we are doing a poor job and how we intend to improve our shortcomings. 

We need to approach this on two broad fronts. First, school 
administrators must be fiscally accountable . We hear all too often that the 
schools could meet all of the public's expectations with existing resources 
if only proper management techniques were used, yet school needs always out- 
run school funds. Fiscal accountability means educators must be prepared to 
identify priority problems with the direct involvement of the people served. 
The second, and probably more difficult, aspect of this is performance 
accountability . We must have a way of indicating, in a manner the public 
can understand and accept, the quality of our educational programs. The only 
comprehensive system of measuring performance is the statewide testing pro- 
gram which has only served as a format for confusion. Very few communities 
escape those newspaper comparisons of district's scores, state scores and 
national norms- -a 11 of which seem to indicate that for whatever the reason-- 
Johnny can't read. 

Educators must be prepared to provide the interpretation of 
the facts that Dewey talks about rather than allow this directionless con- 
fusion to continue. Further, if the testing system is as many contend, in- 
sufficient, it is the professional's role not to simply reject it out of 
hand but to build on it from other sources to recommend improvements and, 
most importantly, translate its results into understandable terms* 

If we do not accept this challenge, the resistance to support- 
ing our schools will grow, along with more prescriptive mandates from the 
state level like those dealing with class size, administrator/ teacher ratios 
and use of funds. It is, therefore, our responsibility to get busy and exact 
the leadership which the public expects so that the desire for accountability 
will be satisfied through better understanding* 
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Research Executive 
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Edward G. Begle 
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ACCOUNTABILITY IN MATHEMATICS LABORATORIES 

Viggo P. Hansen 
Professor of Education 
San Fernando Valley State College 



One of the current fads in mathematics education is the math 
lab. However the term math lab defies description. Schools that have 
these labs generally developed them on the supposition that mathematics 
learning will increase along with student attitudes toward the subject. 
Theories underlying math labs are difficult to articulate and under- 
stand. Furthermore they are seldom supported by any substantive research. 
Also there has been little teacher training and/or preparation cn how to 
use a lab and evaluate its effectiveness. 

This paper is devoted to discussing the results found in a 
math lab program developed at San Fernando Valley State College in con- 
junction with Los Angeles Unified School District. This program was 
begun under the auspices of Trainers of Teacher Trainers in 1969. The 
objectives for the math lab were: 

LONG RANGE OBJECTIVES: 

1. to develop a multi-component mathematics laboratory 
for training future mathematics teachers. 

2. to explore new teaching/learning strategies in mathe- 
matics education through a laboratory environment. 

3. to evaluate the effectiveness of the various laboratory 

components relative to: (1) master teachers , (2) student 

teachers, (3) Junior High students, and (4) college 
personnel „ 

4. to create a computer* monitoring program for scheduling 
students to various laboratory activities depending on 
their needs . 

5. to evaluate the effects the laboratory has on the San 
Fernando Valley community and the Los Angeles Public 
School system. 
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SPECIFIC SHORT RANGE OBJECTIVES: 



1. to develop and define the mathematics laboratory as a 
physical plant. 

2. to schedule all Junior High mathematics classes in the 
laboratory for orientation sessions , about 1500 students. 

3. to schedule selected Junior High mathematics classes 
for intensive instruction on a regular basis in the 
laboratory . 

4. to select and train master teachers from Patrick Henry 
to use the laboratory facilities for their classes. 

5. to select and schedule S.F.V.S.C. student teachers to 
teach their classes in the laboratory. 

6. to develop curriculum materials compatable with both 
the existing mathematics program and a laboratory 
approach to teaching mathematics . This will include 
preparing laboratory lessons, unit plans, and activity 
kits. 

7. to evaluate the effectiveness of the. various components 
and equipment in the mathematics laboratory. Specifically 
to evaluate: 

a. Laboratory Activity Sheets 

b. Computer Hardware 

c. Selected Mathematics Readings 

d. Filmstrips, Film Loops and Transparencies 

e. Audio Tapes and Listening Posts 

f. Student Projects - Individual and Group 

g . Mathematics Games 

8. to train and evaluate teaching strategies in the mathe- 
matics laboratory. Specifically to train teachers in: 

a. Team Teaching - teams to be composed of college 
professors, master teachers, student teachers, 
and Junior High students. 

b. Team Learning - students working as teams 
using various laboratory facilities. 

c. Large Group Instruction using auditorium facil- 
ities . 

9. to use the time-share computer facilities to help master 
and student teachers in record keeping and to prescribe 
learning activities. 



10. to give all future mathematics teacher instruction on 

a laboratory approach while enrolled in methods courses 
at San Fernando Valley State College. 

11. to establish an Advisory Committee of community, college, 
and public school personnel. 

A few of the findings that will be reported on at this 
session include: 

1. Seventh graders gained 2.5 years in computation skills, 
but only .9 years in arithmetic reasoning. However, 
some individual students had extreme gains which in- 
dicate that this lab is highly beneficial for some 
students. The question now is, what kind of student. 

2. Student teachers who taught in the mathematics labor- 
atory expressed differences in the time required for 
lesson planning as compared with traditional lecture- 
discussion session, etc. 

3. Master teachers reactions tended to be favorable, how- 
ever they have many specific concerns regarding the 
amount of time students should spend in a laboratory, 
role of textbooks, etc. 

4. Student attitudes as measured by a modified Hoyt- 
MacEachern Attitude Inventory indicated a loss of 
interest in mathematics for many students as a result 
of the laboratory experience. 

The implications of the findings from this TTT program seem to 
indicate that a math lab is indeed a highly complex learning environment. 
Accountability must be defined in a very specific terms relatxve to each 
student, the teacher, and to the particular device and/or laboratory 
lessons being used. The role of the teacher in mathematics laboratories 
is considerably different from that of a traditional lecturer. The teacher 
training program at San Fernando Valley State College is being altered to 

incorporate the findings from this reasearch. 

We are now beginning our third year in this project. It now 
includes two junior high schools and one senior high school, a total of 



about 5000 students . 
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SOME COMMENTS ON ACCOUNTABILITY 
IN MATHEMATICS EDUCATION 

Edward Griffith Begle 



(Edward Griffith Begle is Professor of Mathe- 
matics Education at Stanford University and 
Director of The School Mathematics Study Group. 
He received both his B.A. and M.A. degrees from 
University of Michigan, and his Doctorate degree 
from Princeton University.) 



The word "accountability", when used in discussions of educa- 
tion, often means that some person (e.g., teacher) or institution (e.g.,. 
school) or educational procedure (e.g., individually prescribed instruction) 
is assigned the responsibility for a specified amount of student progress to- 
ward specified goals. This is in contrast to assigning the blame entirely to 
the student in case of failure. 

Goals and objectives can be specified and tested for much more 

sophistic at edly now than ten years ago. 

However, in practice we seem to avoid careful examination of 
objectives. Instead, we use existing tests and hence the objectives implicit 
in them. This is bad for two reasons. Many of these tests are out of date 
in view of the major changes in math curriculum in the last decade. Also, 
they were designed for evaluation of individuals, rather than teachers, 
schools, or programs. In particular, standardized tests do not cover the 
range of objectives any careful review is likely to provide. 

During the past decade, a number of schemes for examining ob- 
jectives have been suggested. Each of these schemes takes the view that the. 
objectives of mathematics education are multivariate. Generally, they classi 
fy objectives along two dimensions, content and cognitive level. Thus, for 
example, SMSG uses this matrix: 




To state the objectives of a particular course, or sequence of 
of courses, the relevant boxes are specified. 
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Of course, in practice each column is split into finer subdivi- 
sions. For example, arithmetic could be split into whole numbers and frac- 
tions, and of course there are obvious ways of further subdividing. The finer 
the subdivisions, the more specific the objectives. 

Once the objectives of a particular course of study have been 
decided on, then it is necessary to construct instruments to measure progress 
toward these objectives. This is done by constructing a subtest for each of 
the boxes relevant to our objectives and putting these subtests together to 
form our evaluation test. Naturally, each subtest needs to satisfy the usual 
criteria: internal reliability, discriminating power, etc. This generally 

requires a dozen or more items , 'which makes it clear that current standardized 
tests are insufficient. They do not contain enough items to take care of all 
boxes . 

You will now complain that such a test as described above is 
too lengthy, that it would take more time for a student to work through it 
than is available. But now we must remember that we are not interested in as- 
sessing the success of individual students, but rather of teachers, schools, 
or instructional programs. For these, it is not necessary to administer the 
entire test to each student. Instead, each subtest can be administered to a 
randomly selected subset of the students. The success of the teacher, school, 
or program can now be estimated from the results obtained with the random sub- 
sets. This procedure, called "item sampling," has been known for some time, 
but does -not seem to have been widely used. 

Evidence that these procedures are not fanciful dreams comes 
from the fact that they were followed successfully by the National Longitudin- 
al Study of Mathematical Abilities and from the fact that the California State 
Department of Education has commissioned tests of this kind for evaluating the 
State program at grade levels three, six, and eight. 

To summarize, in two important aspects of accountability, set- 
ting clear, sensible goals and using appropriate measuring instruments, we are 
potentially in good shape in mathematics education. We can do a good job 

along these lines if we want to. 

But let me turn now to another and gloomier picture. Starting 
in the fall of 1962, the School Mathematics Study Group conducted a long term 
study of mathematics achievement. Several tens of thousands of students were 
involved, starting at grade levels four, seven, and ten. These students were 
tested extensively each fall and each spring. In addition, much information 
was gathered about the students' families, schools, communities, and teachers. 

The teacher information was collected in order to follow up the 
vast number of studies previously carried out on teacher effectiveness. It 
included not only the usual background information on education, years of ex- 
perience, am ount of recent in-service training, etc., but also attitudes to- 
ward mathematics, toward teaching, toward students, etc. 

Only recently have we had time to analyze any of this teacher 
data. We computed an effectiveness score for each of the teachers involved 
in this study during the 1962-65 school year. The effectiveness score was 
based on the achievement of the teacher's students during the year, with ini- 
tial achievement and mental ability factored out. 

We were not surprised to find that there was considerable vari- 
ance in these effectiveness scores. This was true for both male and female 
teachers, for each of the three grade levels, and for effectiveness measured 
by student achievement both in computation and in understanding of mathemat- 
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ical concepts. . . 

In view of past research, we were not surprised, although we 

were disappointed since we had high hopes for our new measures of teacher at- 
titudes, to find that none of the teacher information we had collected was 
able to account for much of the variance in effectiveness. 

All of this is merely to provide background for our next analy- 
sis. We found that a substantial number of teachers who had been teaching our 
fourth grade students in 1962-65 were teaching some of our fifth grade stu- 
dents in 1965-64. Similarly, a number of the 1962-65 seventh grade teachers 
in our study ;were also involved at the eighth grade level m 1965-64. 

We thus had the opportunity to compare the effectiveness of 
teachers in one year with their effectiveness in the following year. The re- 
sults surprised us. The correlations between Year 1 effectiveness and Year 2 
effectiveness were uniformly low. If a teacher is effective one year we 
should not expect that he will automatically be effective the next year. 

The implications of these findings for accountability in mathe- 
matics education are twofold. First, since the effectiveness of teanhers is 
not a stable trait, it is necessary to measure it frequently. What is worse 
is that if a teacher is not performing at the level we wish, we do not have 
any suggestions to offer as to how he might increase his effectiveness. At 
the s ame time we have no grounds for getting rid of the teacher. Poor per- 
formance last year does not imply poor performance this year. 
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